Elicitation and Evaluation of Statistical Forecasts

This paper studies mechanisms for eliciting and evaluating statistical forecasts. Nature draws a state at random from a given state space, according to some distribution p. Prior to Nature’s move, an expert, who knows p, provides a forecast for a given statistic of p. The mechanism defines the expert’s payoff as a function of the forecast and the subsequently realized state. When the statistic is continuous with a continuum of values, I show that the payoffs that provide strict incentives to the expert exist if and only if the statistic partitions the set of distributions into convex subsets. When the underlying state space is finite, and the statistic is discrete with a finite number of values, these payoffs exist if and only if the partition forms a linear cross-section of a Voronoi diagram—a stronger condition than convexity. In both cases, the payoffs can be fully characterized essentially as weighted averages of base functions. I then analyze mechanisms with stronger properties. Finally I describe several applications. ∗Stanford University Graduate School of Business; email address: [email protected]. †I am deeply indebted to David Pennock and Yoav Shoham for their guidance and invaluable advice. I am also very grateful to Yahoo! Research, where I developed the early version of this work. I thank Google Research and the National Science Foundation (grant CCF-1101209) for their generous support. I am very grateful to Jeremy Bulow, Peter Cramton, Yossi Feinberg, Drew Fudenberg, David Kreps, John Langford, Mohammad Mahdian, Roger Myerson, Michael Ostrovsky, Philip Reny, Alvin Roth, Hugo Sonnenschein, Andrzej Skrzypacz, Balázs Szentes and Robert Wilson for their insightful comments. I thank the participants of the economics, computer science and operations research seminars at Chicago, Columbia GSB, Ecole Polytechnique, Harvard, MIT Sloan, NYU Stern, Stanford GSB, UC Berkeley and Wharton.